Neural networks can be used to analyze images for a variety of purposes. For example, some neural networks can examine images in order to identify objects depicted in the images. The neural networks can be trained to detect various objects in images by providing the neural networks with labeled training images. The labeled training images include images having known objects depicted in the images, with each pixel in the labeled training images identified according to what object or type of object the pixel at least partially represents.
Additionally, complex neural networks pre-trained on large data sets are available. For example, the complex neural networks may be configured to classify images in over 1000 categories. The complex neural networks are proven to be good initialization of other models and can be transferred via fine tuning techniques for many other tasks. However, the complex-performing networks are usually very wide (i.e., many neurons per layer) and deep (i.e., many layers) involving numerous parameters. The parameters represent a different number of weights representing the over 1000 categories the complex neural network can classify. Due to a large number of computations, such complex networks can result in very time consuming operations at inference time. Additionally, having large amounts of parameters or categories of the complex neural network requires a larger memory to store the different parameters.